Development of a Task Oriented Performance Standard Evaluation Program

...................................................................................................................................2 TABLE OF CONTENTS..............................................................................................................4 INTRODUCTION..........................................................................................................................5 BACKGROUND AND SIGNIFICANCE....................................................................................5 LITERATURE REVIEW.............................................................................................................8 PROCEDURES............................................................................................................................11 RESULTS.....................................................................................................................................15 DISCUSSION...............................................................................................................................19 RECOMMENDATIONS.............................................................................................................22 REFERENCES.............................................................................................................................24 APPENDIX A (Performance Standards)...................................................................................26 APPENDIX B (Performance Based Evaluation Check Sheet)................................................29 5 INTRODUCTION The Idaho Falls Fire Department places a high priority on training. An important part of our training program is the practice of engine company evolutions. Department members are expected to participate in these evolutions to maintain basic fire combat skills. The problem is that the Idaho Falls Fire Department does not have a systematic program to evaluate personnel with regards to engine company or individual task performance standards. The purpose of this research project is to develop a program of evaluating the performance of all engine companies and individuals in standard evolutions, tasks, and department requirements. Action research was used in an effort to develop such a program. The following research questions had to be answered: 1. How can the Idaho Falls Fire Department evaluate the ability of individuals to perform necessary tasks at a level of acceptable fire service standards? 2. How can the Idaho Falls Fire Department evaluate the ability of engine companies to perform necessary tasks at a level of acceptable fire service standards? 3. What format would be effective in implementing a performance based evaluation system? 4. What method of documentation can be implemented to increase the effectiveness of a performance based evaluation system? 6 BACKGROUND AND SIGNIFICANCE The Idaho Falls Fire Department, hereafter referred to as the department, has historically attempted to introduce programs designed to confirm competency. In the early 1980s the department adopted what was then termed a physical agility test. All department members, on an annual basis, were required to perform a series of tasks designed to simulate actual fire combat tasks such as donning a self contained breathing apparatus, advancing a hose line, and climbing a ladder. The annual test did manage to confirm a department member’s baseline ability to perform fire related tasks, but the program had numerous shortcomings. The test was comprised of two sections, and firefighters were allowed 20 minutes to complete each section. The time allowed proved excessive, as the more motivated and competent firefighters could complete the tasks in around five minutes. Those department members less motivated and/or incompetent could complete the series of tasks, but they used all the allowed time. It was clear to most of the department’s management that the leniency of the physical agility test rendered it a poor indicator of a firefighter’s actual ability to perform in combat situations. As the department’s administration changed, an attempt was made to increase the test’s validity by decreasing the time allowed. This posed no difficulty for the more motivated and competent firefighters, but the less motivated now complained of numerous injuries, so the physical agility test was abandoned. In retrospect the most prominent weakness of the abandoned testing program was not the time factor, but the lack of any connection to a corresponding training program. The department’s training officer emphasized engine company evolutions, but department 7 management made no attempt to mandate any performance level. The more motivated company officers devoted time to practicing the evolutions at the drill field, but the less motivated officers placed a much lower priority on practical training. Management’s failure to mandate training or test for competency created a broad range of competency levels. The success of fire ground operations varied widely depending upon which crew arrived on the scene. The individual firefighters assigned to an unmotivated officer were not provided with the practical drill field experience to assure their effectiveness in an actual fire situation. The department clearly had a need for a program that would encourage all companies and individuals to train and also allow for confirmation of the effectiveness of that training. “A department that establishes performance standards or timed evolutions for engine and ladder companies and then trains utilizing those criteria will be better prepared to handle the varied problems that occur at an incident scene” (Smith, 1996, p. 16). Besides the practical need for a performance standard program there is also the legal need. As the department moved towards computerized record keeping, it became obvious that not only was it important to demonstrate our capabilities, it was also important to be able to document that demonstration of appropriate training. Davis (1991) states that: To ease the threat of lawsuit, the prudent fire manager will investigate and consider any sensible protection that might be available. This is especially true today, with the epidemic of lawsuits. It seems that every mishap can result in a legal action. (p. 56) Davis goes on to add, “Also, the documentation that results from performance standards (often called ‘paper trails’) can provide valuable support when legal scrutiny occurs” (p.56). The purpose of this research project, the development of a program to evaluate the performance of engine companies and individuals, relates to Unit 10: Service Quality/Marketing 8 of the Executive Development course taught at the National Fire Academy. While discussing service quality in that setting, Drucker (1999) claimed “it takes far more energy and work to improve from incompetence to mediocrity than it takes to improve from first rate performance to excellence” ( p. 67). The successful implementation of a program that will encourage training and allow the department to evaluate our members’ task performance should eliminate the incompetence and allow for future excellence. LITERATURE REVIEW Training is one of the most often discussed topics in the fire service. The appraisal and assessment of training has also received attention. Past generations of firefighters measured their proficiency under fire; turning a building into a parking lot earned that fire crew a failing grade. The trend towards a more proactive assessment of efficiency has given the concept of performance standards momentum. Performance standards have been around for some time. In the 1970s and 1980s , there were many articles about them. Some have viewed performance standards as a passing fad, but many universities and other educational facilities have developed the concept of competency-based education, which is essentially performance standards in a new wrapper. This can be the best training to effectively maintain competency and optimum employee performance. (Davis, 1991, p. 56) The first research question posed referred to the evaluation of an individual’s ability to perform tasks. Performance standard evaluations originally focused on the engine company. As the concept has developed, proponents of performance standards have become aware of the need 9 to evaluate the individual within an assessed engine company. Lecuyer (1999) warns “be aware that an individual could score poorly even if the company met the time standard, so do not overlook the individual ratings” ( p. 94). The results of focusing only on company success could cause an individual’s deficiencies to be overlooked. That individual might be compensated for when operating on the assigned crew, but if reassigned to another crew those deficiencies could prove problematic. Warren (1998) addresses the individual assessment: The evaluation phase signals the end of the learning process and verifies that the original learning objectives have been attained. The only way to adequately implement this phase is for the company officer to test the student’s ability to perform a task without assistance. Evaluations can be accomplished through . . . performance exams for psychomotor objectives. (p.52) Evaluating the task performance of an individual working within a company operation requires a standardized program. “Standardization lets members assigned to different units work together” (Smith, 1996, p. 16). The assessment of individuals also calls for special attention on the part of the assessors. “The evaluation process requires multiple graders so that at least two (preferably three) raters grade each individual” (Lecuyer, 1999, p. 92). The second research question referred to the evaluation of the engine company’s ability to perform task oriented evolutions. “As chief, one of your responsibilities should be to empower and encourage your company officers to verify that all training goals have been met. Otherwise, the effectiveness of your training program is merely a guessing game” (Warren, 1998, p. 52). One of the most common methods of evaluating an engine company’s performance is to establish a company standard that can be demonstrated by a performed evolution. “A company 10 standard is a minimum task performance requirement that a company working as a team must meet” (Lecuyer, 1999, p. 89). If performed and based on a consistent set of guidelines, evolutions offer both training and the verification of capability: The entire evolution must be specific and documented. There should be a maximum amount of time to complete an evolution. Using time frames simulates the stress found at the incident scene. It also demands teamwork on the part of all members to assure that the time frames will be met. (Smith, 1996, p. 16) Engine company evolutions are the bare minimum of fire company training, and those same evolutions can be used to verify proficiency. “Although these evolutions and skills require some practice, after time they become second nature” (Ridley, 1999, p. 21). The third research question asks what format might be used for implementation of a performance based evaluation system. Research indicates a strong inclination toward the use of standard operating procedures or standard operating guidelines. “In the fire service, written standards, and standard operating procedures and standard operating guidelines, are the norm. These standards define or describe accepted practices” (Rukavina, 1998, p. 28). The basic purpose of the implementation of a program to evaluate task oriented performance is to achieve an acceptable level of overall performance. To reach the desired level of consistency, the department needs to document the program and adopt that program as part of the department’s administrative guidelines. “Rules and procedures are a vital part of control and are an essential component of the management process. In any organization, control is necessary to minimize risk and ensure predictable outcomes during standard operations” (Cook, 1999, p. 108). In the unfortunate event of a fire ground injury or death, the department needs to have 11 the ability to verify consistent and competent operations. The standard operating procedure or standard operating guideline format would allow for that verification. “If a fire department has its own rules, regulations, or standard operating procedures, they are what a court would examine to find evidence of a standard or behavior or care” (Bachtler & Brennan, 1995, p. 168). The fourth and final research question asks what method of documentation could be used to increase the effectiveness of a performance based evaluation system. Research reveals that this documentation could be part and parcel with the chosen format. If the program is implemented in a standard operating procedure or standard operating guideline format, the documentation of the program will exist as part of the department’s administrative guidelines. The determination of how to document the actual performance of the exercise should be part of the procedure or guideline. A performance evaluation program should promote training, determine the effectiveness of that training, and document the department’s capability to perform fire combat tasks at an acceptable standard. This documentation should establish both the competence of all department companies and the ability of all individuals. As a public safety organization the department has an obligation to demonstrate competence. Unfortunately, that competence might have to be verified in a post-incident setting, as previously discussed. Effective documentation of both the adopted standard and the demonstrated performance is essential. “Standards are a valuable tool for organizational advancement, but adopting a standard leaves deep footprints. Make sure that your department is fully aware of a standard’s requirements, and build training on those requirements into your department’s continuing-education program” (Rukavina, 1998, p. 30).